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Foreword 
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School’s  Acousdcel  Ooeenognyhy  kssson  wosk  was  conducted  under  the  6.3  Manpower, 
FstsoomI,  aad  Tkaintng  Advanced  Technical  Development  hogiam  Etanent  0603707N 
(WodE  Unit  0637Q7N1233SJM001)  and  the  the  6Jt  Mission  Support  Technolosy 
Piofram  Element  0602233N.  (Penotmel  and  TYaifiing  Technology  Block,  NP2A  Work 
Unh  06G2233NJIM33T23.08)  wfaiefa  was  sponsored  by  the  Chief  of  Naval  Research 
(Gode-34X  The  fonl  of  this  study  was  to  e^^duaie  die  lesson’s  instmctiooal  design  md 
developmett,  modvwional  characteristics,  and  effectiveness. 

The  lecooBaiendatiom  provided  in  this  report  ate  intended  foruse  by  (Xficer  in 
Chaife,  Avionics  Depemnent,  Naval  Air  Tedmica* "  -nog  Center. 

Any  qnearions  regarding  this  report  should  be  >4t  to  Dr.  John  Ellis, 
Instnictkaial  Sdenoes  Division,  (619)  553  9273  or  DS  .  553  9273. 


J.  C.  McLACTILaN 
Dhector,  Tiiining  Research  Dq)aiiiiiu)< 


Summary 


Background  and  Problem 

Job  requirements  for  Navy  system  operators  include  computation,  analysis, 
classification,  interpretation,  decision  maldng,  and  communication.  Technical 
competency  in  these  types  of  depends  upon  the  acquisition  and  retention  of  the 
organized/systematiz^  technical  knowledge  and  skills  underlying  performance. 
Unfommately,  there  is  evidence  that  these  types  of  tasks  may  not  be  well  taught  or 
remembered. 

Recently,  iimovative  computer-based  instructional  approaches  have  been  developed 
to  teach  the  complex  cognitive  concepts  and  skills  involved  in  aviation  based  anti- 
subrruurine  warfare  (ASW).  Specificidly,  the  Interactive  Multisensor  Arudysis  Trainer 
(IMAT)  has  been  introduced  into  the  Aviation  Antisubmarine  Warfare  Operator  (AW) 
"A"  school  and  the  curriculum  has  been  revised  to  accorrmxxlate  the  add^  instructional 
capabilities.  The  IMAT  was  designed  to  facilitate  learning  by  graphically  elaborating 
complex  cause  and  effect  relationships,  visibly  depicting  invisible  phenonmena,  and 
IHOviding  a  mission  oriented  context  for  demonstrations,  examples,  and  practice  during 
die  course  of  instruction. 


Objectives 

The  objectives  of  this  effort  were  to:  (1)  evaluate  die  instructional 
design/development  of  the  IMAT  instruction  and  the  previous  AW  "A"  school  instruction 
which  was  delivered  in  the  standard  lecture  fnmat.  (2)  assess  the  motivational 
characteristics  of  the  IMAT  instructiem,  and  (3)  where  possible,  compare  the 
effectiveness  of  IMAT  instructimi  with  the  previous  AW  "A"  school  instructimi. 


Method 

The  instructional  design  evaluation  used  the  Course  Evaluation  System  (CES)  to 
assess  the  consistency  (match)  among  learning  objectives,  test  items  arid  the  intructional 
presratatitm  and  adequacy  of  the  instructional  presention  for  the  IMAT  course.  For 
comparative  purposes  the  CES  was  used  to  evaluate  the  consistency  of  the  same 
components  fev  die  previous  standard  lecture  course. 

To  assess  the  motivational  charaaeristics  of  the  IMAT  instruction  a  questionnaire 
containing  four  separate  scales  was  given  to  76  students  after  they  completed  the  unit 
exarxL  The  questionnaire  was  designed  to  assess  attention,  relevance,  confidence,  and 
satisfactimi. 

Finally,  IMAT  students  and  students  who  had  taken  the  previous  standard  lecture 
course  were  ctnnpared  on  test  scores  on  selected  items. 
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Results 

Overall  the  instructional  d  gn  evaluation  for  both  the  IMAT  and  the  standard 
lecture  course  vealed  few  serious  problems  or  deficiencies.  Both  cor'ses  were  found  to 
be  superior  to  the  typical  Navy  course.  The  only  major  problem  for  the  IMAT  course 
was  a  high  number  of  untested  objectives  requiring  trainees  to  perform  procedural  and 
cognitive  tasks.  This  was  the  result  of  the  triplication  of  standard  testing  procedures  by 
the  AW  "A"  School  and  was  not  a  product  of  the  IMAT  design  and  development  process. 
The  motivatitmal  questionnaire  analysis  showed  high  positive  scores  on  all  four  scales 
fen-  the  IMAT  course.  When  compared  to  a  recently  resigned  course  on  electricity 
theory  the  IMAT  course  was  supericn’  to  several  different  computer-based  delivery 
strategies.  Finally,  end-of-course  performance  for  the  IMAT  course  and  previous 
standard  lecture  course  did  not  differ  on  selected  test  items.  However,  both  the  IMAT 
and  standard  lecture  course  tests  emphasized  remembering  facts.  The  IMAT  course  is 
longer  and  was  designed  to  teach  more  qualitative  knowledge,  higher  level  concepts,  and 
qiplication  skills  than  the  standard  lecture  course.  Qualitative  knowledge,  concepts  and 
sidlls  were  not  tested  in  either  course  and  the  full  effectiveness  of  the  IMAT  system  can 
not  be  assessed  unless  this  type  of  information  is  tested. 


Recommendations 

1.  The  AW  "A"  school  should  implement  the  changes  and  conections  identified  in 
the  CES  analysis. 

2.  The  AW  "A"  school  should  develop  test  items  to  assess  cognitive  concepts, 
relaticmships,  and  skills  and  revise  the  curmt  Acoustical  Oceanography  tests. 

3.  Evaluation  of  IMAT  training  should  continue  as  IMAT  is  introduced  into 
addititmal  schools.  Pre-post  perfonnance  data  should  be  collected  as  well  as 
motivational  and  instructional  quality  data.  Future  evaluations  should  include  the 
assessment  of  cognitive  skills  uid  qualitative  understanding  in  additkm  to  factual 
knowledge. 

The  Navy  Personnel  Research  and  Develqmient  Center  is  providing  suppon  in 
acc(niq)lishing  these  recommendations  as  part  of  ongoing  project  efforts. 
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Introduction 


Background  and  Problem 

The  learning  and  retention  of  complex  cognitive  knowledge  and  skills  are  essential 
in  today’s  highly  variable  operational  environment  For  example,  sonar  system  operators 
must  detect  classify,  track,  and  target  submarines  under  a  variety  of  conditions  and 
constraints.  Operation  of  sonar  and  other  similar  systems  requires  computation,  aiuilysis, 
classificadmi,  interpretation,  decision  making,  and  communication  skills.  Technical 
competency  in  performing  these  types  of  tasks  depends  upon  the  acquisition  and 
retention  of  the  technical  knowledge  and  skills  required  to  perform  them. 

Unfmtunately,  evidence  from  research  science  education  and  military  technical 
training  indicates  that  these  types  of  tasks  may  not  be  well  taught  or  renumbered  G^ck, 
Parker,  &  Koehler,  1990;  Druckman  &  Bjork,  1991;  McKeithen  &  Reitman,  1981;  Riley, 
1986;  Semb  &  Ellis,  1994;  Wetzel,  Ktmoske,  &  Montague,  1983).  Several  recent 
research  programs  have  attempted  to  address  drese  problems  through  the  development  of 
instructional  strategies  and  methods  designed  to  facilitate  the  learning  of  complex 
knowledge  and  skills  (Q>gniti<xi  and  Technology  Group  at  Vanderbilt,  1990;  Collins, 
Brown,  Sc  Newman,  1989;  Konoske  Sc  Ellis,  1991).  Recently,  an  iimovative  con:q)uter- 
based  instructional  approach  based  on  many  of  these  strategies  has  been  developed  to 
teach  the  ooDq>lex  cognitive  concepts  and  skills  involved  in  aviation  based  anti¬ 
submarine  warfare  (ASW)  (Wetzel-Smith,  1991).  Specifically,  the  Interactive 
Multisensor  Aiudysis  Training  system  (IMAT)  has  been  intn^uced  into  the  Aviation 
Antisubmarine  Warfare  Operator  (AW)  class  ’’A”  school  and  the  curriculum  has  been 
revised  to  accommodate  ^  added  instructional  capabilities.  Strategies  incorporated  in 
the  new  curriculum  include  a  situated,  anchored,  or  contextualized  qrproach  to  te«;hing 
cognitive  concepts  and  skills  (Cognition  and  Technology  Group  at  Vanderbilt,  1990; 
Collins  et  al.,  1989)  and  techniques  that  provide  visually,  verbaUy,  and  textually 
elaborated  explanations  for  conoplex  relationships  and  phenomena.  Such  strategies  have 
been  shown  to  improve  learning  (Konoske  Sc  EUis,  1991;  WhitehiU  Sc  Ellis,  in  process). 
There  is  also  some  evidence  that  the  innovative  methods  being  used  in  the  new  course 
may  differentially  affect  retention  (Semb  Sc  Ellis,  1994). 


Objective 

Tl»  objectives  of  this  effort  were  to:  (1)  evaluate  the  instructional 
designAlevelopment  of  the  IMAT  and  previous  standard  lecture  instructitMi;  (2)  assess  the 
motivati<»al  characteristics  of  the  IMAT  instruction;  and  (3)  where  possible,  compare 
die  ^ectiveness  of  IMAT  instruction  with  the  previous  AW  "A”  school  instruction  using 
performance  on  ccmiparable  test  items. 
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Method 


bistructitmal  Design/Development  Evaluation 

The  Course  Evaluation  System  (CES)  (Ellis,  Knirk,  Taylor,  &  Mc£>onald,  1993) 
was  used  to  evaluate  the  instrucdcmal  design  and  development  of  the  IMAT  and  standard 
lecture  instruction.  The  CES  a.>  esses  the  consistency  (match)  among  learning 
objectives,  test  items,  and  the  instructional  presentation  and  the  adequacy  of  the 
instructional  presentation.  The  CES  was  tq)plied  to  the  objectives,  test  items,  student 
guide,  instructor  guide,  and  instructional  presentation  for  Unit  Seven;  Acoustical 
Oceanography  of  the  Aviation  Antisubmarine  Warfare  Operator  (AW)  CHass  A1  Course. 
For  the  previous  standard  lecture  course  only  the  objectives,  test  items,  student  guide, 
and  instructor  guide  were  evaluated  because  the  instructional  presentation  was  no  longer 
given. 

The  CES  classifies  the  course  objectives  using  a  scheme  that  corresponds  to  the 
general  distinction  between  knowledge  and  skills.  Because  the  AW  course  is  an 
introductory  course  that  emphasizes  acoustical  oceanography  theory,  as  well  as 
applications  of  that  theory  to  anti-submarine  warfare,  the  scheme  was  modified  slightly 
to  include  two  types  of  knowledge  objectives  and  one  type  of  skill  objective.  Course 
objectives  were  classified  in  one  of  the  following  categories: 

Remember  Fact  The  student  must  recall  or  recognize  r^unes,  definitions,  steps  of 
procedures,  formulas  and  terms  in  formulas,  labels  for  graphical  display,  or 
technical  terminology. 

Remember  Qualitative  Informatiou.  The  student  must  recall  or  recognize  cause 
and  effect  relationships,  predictive  and  diagnostic  information,  or  how  and  why 
explanations  of  events  (e.g.,  diurnal  e£f(K;t),  phenomena  (e.g.,  bottom  bounce),  and 
devices  (e.g.,  sensors). 

Use  Procedures  and  Rules.  The  student  must  perform  a  sequence  of  steps  in  a 
certain  order  to  solve  a  problem,  operate  ot  maintain  equipment  calculate  or 
determine  a  value,  or  evaluate  a  scenario. 

For  details  of  the  CES  procedures  and  the  underlying  rationale  see  Ellis,  Knirir,  Taylor, 
and  McDonald  (1987, 1993). 


Motivational  Analysis 

For  the  motivational  analysis  a  questionnaire  based  on  the  Attention-Relevance- 
Qxifidence-Satisfaction  (ARCS)  motivational  assessment  model  developed  by  Keller 
(1992)  was  developed.  The  34  item  questionnaire,  designed  for  application  to  technical 
training,  was  administeted  to  76  AW  "A"  school  students  upon  completion  of  the  Unit 
Seven  comprehensive  test  The  ARCS  questionnaire  assess  four  motivational 
characteristics;  attention,  confidence,  relevance,  and  satisfaction.  The  attention  oriented 
questkms  assess  how  well  the  material  captures  the  interest  of  the  learners  and  stimulates 
their  curiosity  to  learn.  The  relevance  questions  address  how  well  the  materials  meet  the 
needs  and  gcNsls  of  the  learner.  The  confidence  questions  ask  the  students  to  report  on 
their  belies  and  feelings  about  well  they  will  succeed  and  how  much  they  can  control 
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their  own  success.  The  satisfaction  questions  concern  students  feelings  of  reward  and 
accomplishment  and  their  enjoyment  of  the  materials.  The  four  scales  are  scored  from  1 
to  5  with  1  indicating  a  statement  about  the  course  is  “Not  true"  and  a  5  indicating  a 
statement  is  "Very  true."  A  score  of  3  indicates  students  believe  a  statement  is 
"Moderately  true." 


Comparative  Analyses 

Where  possible  the  IMAT  based  instruction  for  Unit  Seven  was  compared  with  the 
standard  lecture  instruction  AW  "A"  school  instruction  on  acoustical  oceanography.  The 
first  comparison  was  to  apply  the  CES  to  as  much  of  the  standard  lecture  course  as 
possible.  As  discussed,  tlus  analysis  included  the  learning  objectives,  test  items, 
instructor  guide  and  student  guide  from  the  previous  standard  lecture  course.  The  results 
of  this  analysis  were  compared  to  the  CES  analysis  on  the  IMAT  course.  The  CES 
analysis  on  the  IMAT  course  was  also  compared  to  data  obtained  in  a  larger  analysis  of 
100  Navy  courses  to  determine  how  the  IMAT  course  compared  to  a  "typical"  Navy 
course. 

The  second  conqraiison  was  designed  to  determine  how  well  the  standard  lecture 
course  students  performed  on  the  IMAT  course  objectives  compared  to  IMAT  course 
students.  To  accomplish  this  analysis  test  items  from  the  standard  lecture  course  that 
tested  the  IMAT  course  objectives  were  identified.  Performance  on  the  standard  lecture 
course  items  that  tested  IMAT  course  objectives  was  compared  to  performance  on  the 
IMAT  course  test  items  for  the  same  objectives. 

The  third  comparison  was  designed  to  determine  how  well  IMAT  course  students 
performed  con^ared  with  the  standard  lecture  course  students  on  the  standard  course 
objectives.  To  accomplish  this  analysis  test  items  from  the  IMAT  course  that  tested  the 
standard  lecture  course  objectives  were  identified  and  test  performance  for  the  two 
student  groups  was  compared. 


Results  and  Discussion 


Instructional  Design/Development  Analysis 

The  top  half  of  Table  1  presents  the  results  of  the  CES  objective  and  test  item 
evaluation  for  the  IMAT  course.  The  major  finding  is  that  a  substantial  number  of 
Remember  Qualitative  information  objectives  and  Use  Procedure  and  Use  Rule 
objectives  are  not  tested.  This  frequently  occurs  in  Navy  schools  (Taylor,  Ellis,  Baldwin, 
1988)  and  can  be  corrected  by  developing  additional  test  items  and  revising  existing 
tests.  This  lack  of  test  coverage  did  not  result  firom  the  IMAT  course  development 
process.  The  tests  for  IMAT  Unit  Seven  were  developed  by  AW  "A"  school  after  the 
unit  was  delivered  to  the  school.  The  tests  developed  at  the  school  were  consistent  with 
traditional  Navy  classroom  tests  which  emphasize  fact  oriented  multiple-choice  questions 
(Taylor  et  al.,  1988).  The  development  of  more  sophisticated  test  items  designed  to  test 
complex  cognitive  relationships  and  skills  is  more  difficult  to  do,  especially  in  a 
multiple-choice  format  Further,  scoring  for  higher  level  items  that  are  not  in  the 
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traditional  multiple-choice  format  is  also  difficult  In  any  case,  to  assess  the 
effectiveness  of  the  IMAT  course  in  teaching  cognitive  concepts,  relationships,  and 
skills,  additional  test  items  need  to  be  developed  and  the  current  tests  need  to  be  modified 
and/OT  revised. 

The  finding  that  a  high  proportion  of  Remember  Fact  and  Remember  Qualitative 
information  objectives  did  not  have  matching  test  items  is  attributable  to  many  objectives 
written  using  the  action  verbs  "state"  and  "define."  These  verbs  require  fiU-in-the-blank, 
short  answer,  or  essay  test  items.  All  the  test  items  for  the  IMAT  course  used  the 
multiple-choice  formaL  This  discrepancy  is  easily  corrected  by  adding  "recognize"  or 
"select  the ..."  to  the  objective  verbs. 

The  top  halt'  of  Table  2  presents  the  objective  and  presentation  consistency  results 
for  the  IMAT  course.  The  major  finding  is  that  for  a  small  percentage  of  each  type  of 
objective  the  practice  component  of  the  presentation  was  either  incomplete  or  not 
present  This  can  be  corrected  by  developing  practice  test  items  for  these  objectives. 

The  CES  adequacy  criteria  and  instructional  effectiveness  checklist  were  also 
applied  to  the  IMAT  course  and  no  deficiencies  were  identified. 


Motivational  Analysis 

The  ARCS  questionnaire  assesses  four  motivational  characteristics;  attention, 
confidence,  relevance,  and  satisfaction.  Each  characteristic  is  rated  from  one  to  five;  not 
turc  to  very  true,  respectively.  The  mean  scores  for  each  scale  for  the  IMAT  course  are: 
Attention  =  3.43,  Confidence  *  3.36,  Relevance  =  3.76,  and  Satisfaction  *  3.48. 

Although  it  was  not  possible  to  collect  ARCS  data  on  the  standard  lecture  course, 
recent  ARCS  data  are  available  for  a  similar  course  employing  computer-based 
instructional  strategies.  ARCS  questionnaires  were  given  to  students  completing  a  lesson 
on  circuits  of  the  Aviation  Electronics  Technician  "A"  School  course.  This  lesson  is 
similar  in  complexity  to  the  Acoustical  Oceanography  lesson.  Four  different  methods 
were  used  to  present  the  circuits  lesson:  (1)  conventional  classroom  instruction,  (2) 
computer-based  drill  and  practice,  (3)  enhanced  con^uter-based  instruction  (CBI) 
(similar  to  IMAT  but  individualized  instead  of  lecture  based),  and  (4)  a  computer-based 
adventure  game  designed  to  teach  concepts  and  problem  solving.  The  three  CBI 
conditions  were  developed  for  a  research  project  and  were  designed  to  be  state  of  the  art 
training.  Special  consideration  was  given  to  the  nootivational  aspects  of  this  traiiung, 
especially  in  the  adventure  game  condition.  Thus,  the  CBI  groups  provide  a  fair 
motivational  comparison  for  classroom  based  IMAT  training.  If  anything,  the  IMAT 
training  is  at  a  disadvantage  because  it  is  not  individualized.  A  one-way  analysis  of 
variance  and  post-hoc  paired  comparisons  compared  on  the  four  groups  from  the 
Aviation  Electronics  course  and  I^T  students.  Students  rated  IMAT  instruction 
significantly  higher  than  classroom  instruction,  drill  CBI,  and  game  CBI  on  relevance, 
confidence,  and  satisfaction.  There  were  no  differences  between  IMAT  and  the  enhanced 
CBI  (the  condition  most  similar  to  IMAT)  on  any  scales  and  no  difference  between 
IMAT  and  the  game  CBI  on  attention.  These  findings  show  that  IMAT  is  as  motivating 
as  similarly  designed  individualized  CBI  and  more  motivating  than  traditional  classroom 
instruction  and  standard  drill  and  practice  CBI.  Table  3  presents  the  means  for  the  RC 
circuit  lessons  and  the  IMAT  lesson. 
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Table  1 

IMAT  and  Previous  Standard  LectureCourse  Objective 
and  Test  Item  Evaluation  Results 


IMAT  Course 

Rating  Category 

Remember  Fact  (%) 

Remember  Qualitative  (%) 

Use(%) 

Objectives  n=54 

Objectives  n=19 

Objective  n-17 

Test  lums  n-138 

Test  Items  n=l  5 

Test  Items  n=17 

Objective  Not  Appropriaie 

0 

0 

0 

Objectives  Not  Essential 

0 

0 

12 

Objectives  Not  Tested 

22 

58 

71 

Test  Items  Don’t  Match  Objective 

62 

33 

12 

Conditions  Don’t  Match 

59 

12 

2 

Standards  Don’t  Match 

0 

0 

0 

Actions  Don’t  Match 

62 

33 

12 

Test  Items  Not  Adequate 

0 

0 

0 

Test  Items  Not  Appropriate 

0 

0 

0 

Previous  Standard  Lecture  Course  | 

Rating  Category 

Remember  Fact  (%) 

Remember  Qualitative  (%) 

Use(%) 

Objectives  n^l3 

Objectives  n-2 

Objective  n-5 

Test  Items  n»47 

Test  Items  n=I2 

Test  Items  n=I4 

Objective  Not  Appropnate 

0 

0 

0 

Objectives  Not  Essmitial 

0 

0 

0 

Objectives  Not  Tested 

8 

0 

0 

Test  Items  Dmi’t  Match  Objective 

100 

100 

100 

Conditions  Don’t  Match 

11 

0 

7 

Standards  Don’t  Match 

100 

100 

100 

Actions  Don’t  Match 

11 

0 

50 

Test  Items  Not  Adequate 

0 

0 

0 

Test  Items  Not  ^qnopriate 

0 

0 

0 
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Table! 

Presentation  Consistencj  for  the  IMAT  and  Previous  Courses 


IMAT  Course 


Required  Presentation  Components 


Objective  Type 

Practice 

Practice 

Statement 

Remembering 

Examples 

Using 

w/Feedback 

w/Feedback 

Remember  Fact 

%  Incomplete 

12 

NA 

NA 

%  Not  present 
n*54 

0 

6 

NA 

NA 

Remember  Qualitative 

%  Incomplete 

0 

16 

NA 

NA 

%  Not  present 
h»19 

0 

5 

NA 

NA 

Use 

%  Incomplete 

0 

0 

0 

0 

%Notiffesent 

n»17 

0 

0 

0 

12 

_ Previous  Course 

Remember  Fact  I  | 


%  Incomplete 
%  Not  present 


0 

0 


15 

15 


NA 

NA 


NA 

NA 


a«i3 _ 

Remember  Qualitative 
%  Incomplete 
%  Not  present 


0 

50 


0 

50 


NA 

NA 


NA 

NA 


«  «2 
Use 


%  Incomplete 
%  Not  present 


0 

20 


0 

0 


0 

20 


0 

0 


n  *5 


note:  NA  «  not  fq;>plicable. 
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Table  3 

ARCS  Means  for  the  RC  Circuit  and  IMAT  Lessons 


GROUP 

Attention 

Relevance 

ConBdence 

Satisfaction 

Classroom 

2.84 

3.05 

3.06 

3.06 

Drill  CBI 

3.02 

3.38 

2.93 

2.74 

Enhanced  CBl 

3.71 

3.54 

3.56 

3.22 

Game  CBI 

3.48 

3.38 

2.66 

3.00 

IMAT 

3.43 

3.76 

3.36 

3.48 

Comparative  Analysis 

Course  Evaluation  System  Comparisons 

The  bottom  halves  of  Tables  1  and  2  present  the  objective  test  item  analysis  and 
presentation  consistency  analysis  for  the  standard  lecture  course.  For  this  course  only 
one  objective  was  not  tested.  However,  none  of  the  objectives  and  test  items  matched 
because  each  objective  required  a  performance  standard  of  no  errors  while  the  course  test 
had  a  passing  standard  of  80%.  This  deficiency  is  correctable  by  changing  the  standard 
for  either  the  objectives  or  the  course  test  Hie  consistency  findings  show  missing 
practice  for  a  small  percentage  of  the  the  Remember  Fact  objectives  and  for  one  of  the 
two  Remember  Qualitative  information  objectives.  Presentation  adequacy  and 
instructional  presentation  checklist  evaluations  could  not  be  performed  on  the  standard 
lecture  course  because  these  analyses  require  classroom  observations  and  the  course  is  no 
longer  given.  Overall  the  findings  for  the  standard  lecture  course  and  the  IMAT  course 
are  similar.  There  are  no  major  consistency  discrepancies  and  no  serious  problems  with 
the  objective  and  test  items  matches.  The  most  serious  discrepancy  is  the  large  number 
of  untested  higher  level  knowledge  and  skill  objectives  in  the  IMAT  course.  However, 
this  finding  is  typical  of  many  Navy  courses.  In  a  CES  evaluation  of  100  Navy  courses 
Taylor  et  aL  (1988)  found  that  almost  50%  of  the  objectives  evaluated  were  not  tested. 
Additionally,  for  those  objectives  that  were  tested  aliix}st  50%  had  test  items  that  did  not 
match.  In  addition,  unlike  the  AW  courses,  over  half  of  the  objectives  and  over  one  third 
of  the  test  items  were  found  to  be  not  appropriate.  In  general,  both  the  IMAT  course  and 
the  standard  lecture  course  are  superior  to  the  typical  Navy  course  based  on  the  CES 
criteria. 

While  the  IMAT  and  standard  lecture  course  are  similar  based  on  the  CES  analysis 
there  are  some  quantifiable  differences  between  them  that  indicate  a  substantial  change  in 
the  instructional  approach  in  the  IMAT  course.  First,  there  is  the  IMAT  trainer,  which 
uses  computer  technology  to  illustrate  and  contextualize  complex  concepts  and  skills. 
Second,  the  total  number  of  objectives  in  the  IMAT  course  (not  counting  lesson  terminal 
objectives)  is  90  versus  20  in  the  standard  lecture  course.  This  is  the  result  of  a  more 
thorough  front-end  analysis  of  the  knowledges  and  skills  that  need  to  be  taught  As  a 
result  the  course  length  for  the  IMAT  course  is  58  classroom  periods  versus  26  periods 
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for  the  standard  lecture  course.  Finally,  there  are  proportionaiely  fewer  Remember  Fact 
and  «ie  Remembc  Qualitative  infcnmation  objectives  in  the  IMAT  course  than  in  the 
star. .  i  lecture  coi-  e.  This  reflects  a  concern  for  teaching  the  complex  cognitive 
concepts  and  relationships  needed  to  perform  in  an  increasingly  technological  work 
environment 

i  est  Performance  Comparisons 

The  end  of  course  tests  for  the  standard  lecture  course  and  IMAT  course  were 
different.  In  order  to  con^aie  performance  two  different  test  item  matches  were 
performed.  First,  test  items  from  the  old  course  were  matched  with  the  IMAT  course 
objectives  and  second,  test  items  from  the  IMAT  course  were  matched  with  the  old 
course  objectives.  For  the  first  comparison  an  overall  score  was  computed  for  the  old  test 
items  that  matched  IMAT  course  objectives.  There  were  62  out  of  a  possible  74  standard 
lecture  course  test  items  that  matched.  The  62  items  tested  22  of  the  90  IMAT  course 
objectives.  The  overall  score  for  the  62  standard  lecture  course  test  items  was  compared 
to  an  overall  score  for  the  70  IMAT  course  test  items  that  tested  the  same  objectives.  The 
overall  score  for  standard  lecture  course  test  items  was  86.65%  compared  to  86.57%  for 
the  EMAT  test  items.  There  were  no  differences  between  the  two  courses  on  this 
measure,  therefore,  an  objective  by  objective  comparison  was  not  performed. 

For  the  second  comparison,  an  overall  score  was  computed  for  the  IMAT  test  items 
that  matched  standard  lecture  course  objectives.  There  w«v  73  out  of  a  possible  168 
IMAT  course  test  items  that  matched.  The  73  items  tested  13  of  the  20  standard  lecture 
course  objectives.  The  overall  score  for  the  73  IMAT  course  test  items  was  compared  to 
an  overall  score  for  the  58  standard  lecture  course  test  items  that  tested  the  same 
objectives.  The  overall  score  for  standard  lecture  course  test  items  was  86.27% 
compared  to  86.42%  for  the  IMAT  test  items.  Overall  there  were  no  differences  between 
the  two  courses  on  this  measure,  therefore,  an  objective  by  objective  cmnparison  was  not 
poformed. 

The  high  overall  scores  and  lack  of  pexfOTmance  differences  between  the  courses  on 
items  testing  the  other  course’s  objectives  indicates  that  each  course  is  effectively 
teaching  the  information  tested.  Recall,  however,  that  both  the  IMAT  and  standi 
lecture  course  tests  emphasized  remembeting  facts.  Recall  also  that  the  IMAT  course  is 
longer  and  was  designed  to  teach  more  qualitative  knowledge,  higher  level  concepts,  and 
application  skills  than  the  standard  lecture  course.  Qualitative  knowledge,  concepts  and 
skills  were  not  tested  in  either  course  and  the  full  effectiveness  of  the  IMAT  system  can 
not  be  assessed  unless  this  type  of  information  is  tested. 


Recommendations 

1.  The  AW  "A"  school  should  implement  the  changes  and  corrections  identified  in 
the  CES  analysis. 

2.  The  AW  "A”  school  should  develop  test  items  to  assess  cognitive  concepts, 
relationships,  and  skills  and  revise  the  current  Acoustical  Oceanography  tests. 

3.  Evaluation  of  IMAT  training  should  continue  as  IMAT  is  introduced  into 
additional  schools.  Pre-post  performance  data  should  be  collected  as  well  as 
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mocivatioiial  and  tnstructionaJ  quality  data.  Future  evaluations  should  include  the 
assessment  (^cognitive  skills  and  qualitative  understanding  in  addition  to  factual 
knowledge. 

The  Navy  Personnel  Research  and  Development  Center  is  {noviding  support  in 
accomplishing  these  recommendations  as  part  of  ongoing  project  efforts. 
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